NPS-AM-14-C1 1 P20R01-070 


PROCEEDINGS 

OF  THE 

ELEVENTH  ANNUAL  ACQUISITION 
RESEARCH  SYMPOSIUM 


THURSDAY  SESSIONS 
VOLUME  II 


AoAs:  Toward  a  More  Rigorous  Determination  of  Scope 

George  Thompson,  Analytic  Services  Inc. 

Jaime  Frittman,  Analytic  Services  Inc. 

John  Yuhas,  Analytic  Services  Inc. 

Published  April  30,  2014 


Approved  for  public  release;  distribution  is  unlimited. 
Prepared  for  the  Naval  Postgraduate  School,  Monterey,  CA  93943. 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


Form  Approved 
OMB  No.  0704-0188 


Report  Documentation  Page 


Public  reporting  burden  for  the  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources,  gathering  and 
maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of  information, 
including  suggestions  for  reducing  this  burden,  to  Washington  Headquarters  Services,  Directorate  for  Information  Operations  and  Reports,  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington 
VA  22202-4302.  Respondents  should  be  aware  that  notwithstanding  any  other  provision  of  law,  no  person  shall  be  subject  to  a  penalty  for  failing  to  comply  with  a  collection  of  information  if  it 
does  not  display  a  currently  valid  OMB  control  number. 


1.  REPORT  DATE 

30  APR  2014 

2.  REPORT  TYPE 

3.  DATES  COVERED 

00-00-2014  to  00-00-2014 

4.  TITLE  AND  SUBTITLE 

Analyses  of  Alternatives:  Toward  a  More  Rigorous  Determination  of 

Scope 

5a.  CONTRACT  NUMBER 

5b.  GRANT  NUMBER 

5c.  PROGRAM  ELEMENT  NUMBER 

6.  AUTHOR(S) 

5d.  PROJECT  NUMBER 

5e.  TASK  NUMBER 

5f.  WORK  UNIT  NUMBER 

7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESS(ES) 

Analytic  Services  Inc, 5275  Leesburg  Pike, Falls  Church, YA, 22041 

8.  PERFORMING  ORGANIZATION 

REPORT  NUMBER 

9.  SPONSORING/MONITORING  AGENCY  NAME(S)  AND  ADDRESS (ES) 

10.  SPONSOR/MONITOR’S  ACRONYM(S) 

11.  SPONSOR/MONITOR’S  REPORT 
NUMBER(S) 


12.  DISTRIBUTION/AVAILABILITY  STATEMENT 

Approved  for  public  release;  distribution  unlimited 

13.  SUPPLEMENTARY  NOTES 

14.  ABSTRACT 

Analyses  of  alternatives  (AoAs)  play  an  important  part  in  the  acquisition  process.  The  act  of  selecting  a  set 
of  alternatives  to  be  compared  in  an  AoA  can  help  decision-makers  understand  and  manage  key 
tradeoffs???or  it  can  prematurely  constrict  or  otherwise  distort  the  solution  trade  space.  It  is  fairly  easy  to 
point  to  completed  AoAs  as  evidence  that  many  of  them  have  been  poorly  scoped.  But  it  is  much  more 
difficult  to  spot  the  problem  in  real  time???  or  to  keep  it  from  occurring  at  all.  This  report  identifies  four 
principles  designed  to  help  minimize  the  occurrence  of  poorly  scoped  AoAs.  These  four  principles  were 
arrived  at  by  applying  formalisms  from  the  disciplines  of  systems  analysis  and  systems  thinking,  in 
combination  with  a  series  of  semistructured  interviews  with  members  of  the  AoA  stakeholder  community 
(consumers  sponsors,  practitioners,  and  critics).  The  principles  may  be  summarized,  in  systems 
terminology,  as  Focus  on  Outputs  and  Think  Backwards;  Start  With  the  Exterior  and  Work  Inwards; 
Apply  Constraints  Carefully;  and  Iterate  and  Reduce  Uncertainty.  The  report  translates  these  principles 
into  practical  terms  that  can  be  understood  and  applied  by  AoA  stakeholders. 


15.  SUBJECT  TERMS 


16.  SECURITY  CLASSIFICATION  OF: 

17.  LIMITATION  OF 
ABSTRACT 

18.  NUMBER 
OF  PAGES 

19a.  NAME  OF 
RESPONSIBLE  PERSON 

a.  REPORT 

unclassified 

b.  ABSTRACT 

unclassified 

c.  THIS  PAGE 

unclassified 

Same  as 
Report  (SAR) 

42 

Standard  Form  298  (Rev.  8-98) 

Prescribed  by  ANSI  Std  Z39-18 


The  research  presented  in  this  report  was  supported  by  the  Acquisition  Research 
Program  of  the  Graduate  School  of  Business  &  Public  Policy  at  the  Naval 
Postgraduate  School. 

To  request  defense  acquisition  research,  to  become  a  research  sponsor,  or  to  print 
additional  copies  of  reports,  please  contact  any  of  the  staff  listed  on  the  Acquisition 
Research  Program  website  (www.acquisitionresearch.net). 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


Panel  20.  Enabling  Affordable  Programs  Through 
Informed  Early  Decisions 


Thursday,  May  15,  2014 

3:30  p.m.  - 
5:00  p.m. 

Chair:  Michael  McGrath,  Vice  President,  Systems  &  Operations  Analysis, 

Analytic  Services  Inc. 

AoAs:  Toward  a  More  Rigorous  Determination  of  Scope 

George  Thompson,  Analytic  Services  Inc. 

Jaime  Frittman,  Analytic  Services  Inc. 

John  Yuhas,  Analytic  Services  Inc. 

Effectiveness  of  Competitive  Prototyping  and  Preliminary  Design  Review 
Prior  to  Milestone  B 

William  Fast,  Naval  Postgraduate  School 

Valuation  of  Capabilities  and  System  Architecture  Options  to  Meet 
Affordability  Requirement 

Ronald  Giachetti,  Naval  Postgraduate  School 

MSi 

TMlBWlTt*  m 


ACQUISITION  RESEARCH  PROGRAM: 
CREATING  SYNERGY  FOR  INFORMED  CHANGE 


-442- 


Analyses  of  Alternatives:  Toward  a  More  Rigorous 
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Abstract 

Analyses  of  alternatives  (AoAs)  play  an  important  part  in  the  acquisition  process.  The  act  of 
selecting  a  set  of  alternatives  to  be  compared  in  an  AoA  can  help  decision-makers 
understand  and  manage  key  tradeoffs — or  it  can  prematurely  constrict  or  otherwise  distort  the 
solution  trade  space.  It  is  fairly  easy  to  point  to  completed  AoAs  as  evidence  that  many  of 
them  have  been  poorly  scoped.  But  it  is  much  more  difficult  to  spot  the  problem  in  real  time — 
or  to  keep  it  from  occurring  at  all. 

This  report  identifies  four  principles  designed  to  help  minimize  the  occurrence  of  poorly 
scoped  AoAs.  These  four  principles  were  arrived  at  by  applying  formalisms  from  the 
disciplines  of  systems  analysis  and  systems  thinking,  in  combination  with  a  series  of  semi- 
structured  interviews  with  members  of  the  AoA  stakeholder  community  (consumers, 
sponsors,  practitioners,  and  critics). 

The  principles  may  be  summarized,  in  systems  terminology,  as  Focus  on  Outputs  and  Think 
Backwards;  Start  With  the  Exterior  and  Work  Inwards;  Apply  Constraints  Carefully;  and 
Iterate  and  Reduce  Uncertainty.  The  report  translates  these  principles  into  practical  terms 
that  can  be  understood  and  applied  by  AoA  stakeholders. 

Introduction 

Background 

Within  the  framework  of  the  Defense  Acquisition  System  (DAS),  an  analysis  of 
alternatives  (AoA)  “assesses]  the  potential  materiel  solutions  to  satisfy  the  capability  need 
documented  in  [an]  approved”  initial  capabilities  document  (ICD;  USD[AT&L],  2008,  §  4.C.5). 
Conducted  ad  hoc  during  the  1970s  and  ‘80s  under  the  label  of  cost  and  operational 
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effectiveness  analysis  (COEA),  this  type  of  assessment  has  been  an  important  feature  of 
the  acquisition  landscape  ever  since  a  major  revision  in  the  Department  of  Defense  (DoD) 
5000-series  regulations  in  1991  (USD[A],  1991;  Balut  et  al.,  2004). 1 

Arguably,  this  emphasis  has  proved  to  be  a  mixed  blessing  over  the  years.  On  the 
one  hand,  the  “formality  and  inevitability”  of  AoAs  has  helped  guard  against  the  potential  for 
premature  selection  of  a  preferred  alternative  (Smith  &  Thompson,  1995,  p.  11).  However, 
the  very  complexity  of  the  process  has  also  made  it  prone  to  start-up  delays,  which,  when 
combined  with  the  pressure  for  timely  decisions,  leads  to  the  seemingly  inevitable  result  that 
“the  time  for  actual  analysis  compresses”  (Smith  &  Thompson,  1995,  p.  14).  Something  has 
to  give,  and  a  common  practice  has  been  to  narrow  the  AoA  scope  (for  example,  by 
constraining  the  set  of  alternatives). 

Complicating  the  problem  of  AoA  scoping  has  been  a  steady  pressure  to  make  the 
rigorous  comparison  of  alternative  solutions  a  mandatory  step  that  occurs  earlier  and  earlier 
in  the  acquisition  timeline.  The  1996  revision  of  the  DoD  5000  series  formalized  the 
requirement  for  AoAs  to  be  performed  before  Milestone  B  (program  initiation;  USD[A&T], 
1996a;  1996b).  And  the  2003  revision — which  was  accompanied  by  the  introduction  of  the 
Joint  Capabilities  Integration  and  Development  System  (JCIDS) — included,  for  the  first  time, 
a  requirement  for  AoAs  to  be  conducted  during  the  Materiel  Solution  Analysis  phase,  that  is, 
prior  to  Milestone  A  (USD[AT&L],  2003a;  2003b).2  The  earlier  the  AoA,  the  wider  the  set  of 
potential  alternatives  (see  Figure  1)  and  the  more  disparate  they  are.  As  a  result,  it  is  more 
difficult  to  decide  which  of  them  should  be  considered  in  the  assessment. 


1  DoD  5000. 2-M  (USD[A],  199)  contains  detailed  guidance  for  conducting  and  documenting  these  analyses. 

Balut  et  al.  (2004,  p.  19)  presented  a  chronology  from  a  cost-analysis  perspective  and  emphasized  the 
importance  of  the  1991  revision. 

2  The  requirement  for  functional  area,  functional  need,  and  functional  solution  analysis  spelled  out  in  the  CJCS 
3170  series  of  instructions  in  2003  can  be  interpreted  as  a  desire  to  push  this  process  even  further  to  the  left,  so 
to  speak,  on  the  timeline.  Interestingly,  Smith  and  Thompson  (1995)  had  eight  years  earlier  argued  the  case  for 
“putting]  the  initial  [AoA]  into  the  time  normally  reserved  for  ‘requirements  analysis’” — or,  at  the  least,  “importing] 
some  of  the  attributes  of  [an  AoA]  into  the  requirements  analysis  process  that  already  takes  place”  (p.  15). 
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Figure  1.  AoA  Scoping  in  the  Context  of  DAS  Phases  and  Milestones 

Evidence  of  a  Problem 

Unfortunately,  there  is  evidence  that  the  AoAs  have  not  always  been  properly 
scoped.  In  2009,  the  Government  Accountability  Office  (GAO)  reviewed  32  major  DoD 
acquisition  programs,  22  of  which  had  included  a  formal  AoA.  Of  these,  13  AoAs — more 
than  half — were  characterized  as  overly  “narrow  [in]  scope”  in  the  sense  that  they  “focused 
on  a  limited  number  of  alternatives”  (GAO,  2009,  pp.  7,  9).  Moreover,  there  was  a  strong 
correlation  between  AoAs  with  a  narrowly  scoped  set  of  alternatives  and  the  occurrence  of 
cost  growth  in  the  ensuing  programs  (GAO,  2009,  pp.  10-12). 

More  anecdotally,  it  has  been  common  in  the  authors’  experience  to  hear  AoAs 
described  as  being  scoped  around  a  predetermined  solution.  (“By  the  time  you  reach  the 
Materiel  Development  Decision  [MDD],  someone  has  already  decided  what  the  solution  is 
going  to  be”  [see  Husband  &  Kaspersen,  2012,  p.  9;  Smith  &  Thompson,  1995,  p.  11]).  To 
the  extent  this  is  true,  the  AoA  becomes  a  square-filling  exercise  or,  at  best,  a  process  of 
exploring  minor  embellishments  around  a  particular  type  of  technology,  weapon,  platform,  or 
piece  of  equipment.  The  potential  to  conduct  a  wide-open  look  at  alternative  technology 
solutions  is  lost. 

2009  WSARA  and  Other  Reforms 

The  Weapon  Systems  Acquisition  Reform  Act  (WSARA)  of  2009  included  important 
changes  designed  in  part  to  address  this  problem.  A  major  change  in  governance  was  the 
designation  of  the  director,  cost  assessment  and  program  evaluation  (DCAPE) — a  Senate- 
confirmed  position — as  responsible  not  only  for  formulating  AoA  guidance  but  also  for  the 
performance  of  the  analysis.  This  responsibility  includes  the  authority  to  reject  or  redirect  an 
AoA. 


WSARA  (2009)  specifies  that  the  AoA  study  guidance  promulgated  by  the  DCAPE 
should  ensure  the  “full  consideration  of  possible  trade-offs  among  cost,  schedule,  and 
performance  objectives.”  By  implication,  “proper  consideration  of  tradeoffs  includes  ensuring 
...  that  a  range  of  sufficiently  different  alternatives  are  examined”  (Husband  &  Kaspersen, 
2012,  p.  9). 

To  help  discharge  these  responsibilities  during  AoA  execution,  the  DCAPE  typically 
establishes  a  senior  advisory  group  (SAG)  that  includes  the  office  of  the  under  secretary  of 
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defense  for  acquisition,  technology,  and  ogistics  (AT&L),  as  well  as  Joint  Staff  and  Service 
representatives.  In  theory,  the  SAG  provides  a  mechanism  for  users,  analysts,  and 
overseers  to  adjust  AoA  scope  during  execution  if  necessary. 

Implementation  of  WSARA  has  been  accompanied  by  a  push  to  shorten  AoA 
timelines.  The  typical  duration  of  a  pre-WSARA  AoA  has  been  variously  estimated  at  16-24 
months.3  In  contrast,  several  stakeholders  interviewed  for  this  project  cited  6-9  months  as 
the  current  goal.  This  shortening  of  AoA  timelines  will  have  significant  impacts — both 
positive  and  negative — on  the  problem  of  AoA  scope;  these  are  discussed  further  in  the 
section  titled  Why  Are  Some  AoAs  Poorly  Scoped? 

The  number  of  completed  AoAs  conducted  entirely  within  the  post-WSARA 
timeframe  is  modest.  The  consensus  within  the  acquisition  community  seems  to  be  that  it  is 
still  too  soon  to  say  exactly  how  the  formal  realignment  of  AoA  responsibilities,  the 
introduction  of  the  SAG,  and  the  push  for  dramatically  shorter  timelines  have  impacted  AoA 
scoping. 

Purpose 

This  project  aimed  to  identify  a  set  of  guiding  principles  that  can  be  applied  by  AoA 
practitioners  and  sponsors  to  reduce  the  incidence  of  improperly  scoped  analyses.  In 
particular,  we  wanted  to  develop  principles  that  were  both  (a)  grounded  in  the  discipline  of 
systems  thinking  (see  Edson,  2008)  and  (b)  practical,  in  the  sense  that  they  can  be  readily 
understood  and  applied. 

In  striving  for  more  rigor,  the  authors  are  not  proposing  to  remove  all  subjectivity  or 
sense  of  “art”  from  the  job  of  scoping  an  AoA.  Rather,  we  sought  a  middle  ground  in  which 
decisions  regarding  AoA  scope  can  be  influenced  by  considerations  more  systematic  and 
rigorous  than  those  arising  solely  from  political,  bureaucratic,  and/or  programmatic 
pressures. 

Limitations 

This  paper  is  not  intended  as  a  guide  to  conducting  AoAs  or  related  studies.  Many 
such  guides  do  exist,  and  the  findings  and  recommendations  presented  here  are  meant  to 
be  understood  and  applied  in  the  context  of  those  references.4  Similarly,  it  is  assumed  that 
the  reader  already  has  a  basic  familiarity  with  the  DAS  and  JCIDS  processes. 

In  this  paper,  the  term  scope  is  associated  with  the  set  of  alternatives.  To  be  sure, 
there  are  other  dimensions  of  scope,  such  as  the  set  of  scenarios,  assumptions,  or  study 
constraints.  We  considered  these  other  dimensions  only  insofar  as  they  shape  decisions 
about  the  set  of  alternatives. 


3  Kowal,  in  2009,  gave  an  average  of  two  years  with  up  to  six  years  in  some  cases.  At  the  other  extreme, 
information  obtained  from  stakeholder  interviews  cited  an  average  of  16  months.  By  way  of  corroboration,  Lihani 
(2011)  cited  USAF  Office  of  Aerospace  Studies  data  on  27  AoAs  conducted  from  2000  to  2008;  the  average 
length  was  20  months.  Of  the  22  AoAs  considered  in  GAO  09-665  (2009),  the  majority  took  between  13  and  30 
months. 

4  For  example,  see  USAF  Office  of  Aerospace  Studies  (2010),  AoA  Handbook ;  Joint  Staff  Force  Structure 
(2009),  CBA  User’s  Guide;  and  DAU  (2013),  Defense  Acquisition  Guidebook,  section  3.3  (Analysis  of 
Alternatives). 
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Factors  such  as  organizational  roles  and  responsibilities,  institutional  culture,  and 
even  the  predilections  of  individual  decision-makers  and  executives  all  contribute  to  the 
incidence  of  improper  AoA  scoping.  While  acknowledging  the  importance  of  such  factors, 
this  paper’s  recommendations  do  not  address  them  directly. 

A  Note  on  Terminology 

The  word  system  can  have  many  connotations.  (Within  the  opening  pages  of  this 
paper,  for  instance,  we  have  already  referred  to  the  “acquisition  system,”  the  Joint 
Capabilities  Integration  and  Development  System,  and  the  “weapon  systems”  that  may 
represent  alternative  solutions  to  a  capability  need.)  For  clarity,  the  text  indicates  the  specific 
type  of  system  that  is  meant  in  each  case.  In  particular,  the  ships,  aircraft,  vehicles,  sensors, 
communications  equipment,  computers,  and  so  forth  that  may  be  the  objects  of  study  in  an 
AoA  will  be  collectively  referred  to  as  combat  systems  or  combat  support  systems  (C/CSSs). 

Approach 

The  first  step  in  applying  a  systems  thinking  approach  is  to  understand  the  nature  of 
the  problem  (see  the  following  section).  Only  then  can  we  explore  various  system  views  (see 
section  titled  Applying  a  Systems  View)  and  formulate  the  resulting  insights  as  a  set  of 
recommendations  (see  Recommendations  and  Observations  section)  aimed  at  improving 
the  practice  of  determining  AoA  scope. 

This  approach  entailed  two  broad  methods  of  inquiry  and  analysis.  The  first  method 
was  to  leverage  the  existing  body  of  knowledge  by  reviewing  available  documents  (DoD  and 
Service  instructions,  guidance,  and  handbooks;  published  reports  and  articles;  and  other 
publicly  available  materials)  and  conducting  interviews  with  stakeholders  representing 
various  roles  (that  is,  identifying  the  capability  needs  being  addressed  in  AoAs;  overseeing 
the  conduct  of  AoAs;  performing  AoAs;  and  reviewing  and  critiquing  completed  AoAs).  The 
second  method  was  to  apply  formal  systems  thinking  paradigms.  These  two  methods  were 
applied  in  concert  and  iteratively:  more  than  once  we  used  systems  thinking  formalisms  to 
gain  insights  into  an  issue  identified  during  interviews,  and  then  used  subsequent  interviews 
to  test  the  validity  of  those  insights. 

All  interviews  were  conducted  on  a  strict  non-attribution  basis.  Statements  derived 
from  interviews  are  substantiated  in  this  report  only  to  the  extent  of  specifying  that  a 
particular  view  was  expressed  by  one,  some,  several,  most,  or  all  interviewees. 

The  research  and  analysis  presented  herein  was  conducted  from  May  2013  through 
January  2014. 

Understanding  the  Problem 

So  far,  we  have  not  specified  exactly  what  it  means  to  say  that  an  AoA  is  improperly 
scoped.  The  following  section  addresses  this  key  point,  followed  by  a  presentation  of  some 
important  causes  of  improper  AoA  scoping,  as  identified  during  stakeholder  interviews. 

What  Is  a  “Poorly  Scoped”  AoA? 

Establishing  consensus  on  this  question  proved  surprisingly  difficult.  Almost  all 
interviewees,  when  asked  the  question  directly,  preferred  to  skip  over  it  and  proceed  straight 
to  a  discussion  of  causes.  In  some  cases,  that  discussion  made  it  clear  how  the  interviewee 
was  defining  the  problem;  in  other  cases,  it  did  not. 

Guided  as  we  were  by  a  systems  viewpoint — including  the  notion  that  alternatives 
represent  a  set  that  has  a  boundary  (see  Analysis  of  Selected  Conceptagon  Triplets 
section) — we  consolidated  interviewees’  implied  constructs  in  the  form  of  two  definitions 
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based  on  the  properties  of  boundaries.  First,  an  AoA  could  be  “too  narrowly  scoped”  if  its 
boundary  excludes  alternatives  that  should  be  considered.  Second,  an  AoA  could  be  “too 
broadly  scoped”  if  it  includes  alternatives  that  should  not  be  considered.5 

These  definitions,  although  helpful,  still  contain  more  than  a  little  question  begging. 
On  what  basis,  can  we  say  that  an  alternative  “should”  or  “should  not”  be  inside  the  AoA 
boundary?  Here,  it  is  important  to  remember  that  the  purpose  of  an  AoA  is  to  inform  a 
decision  about  the  materiel  solution  to  a  capability  need.  Thus,  poor  scoping  is  that  which 
contributes  to  a  poor  decision  (for  example,  by  excluding  a  viable  alternative  that  might  have 
been  judged  preferable  under  some  reasonable  weighing  of  costs  and  benefits)6  or 
otherwise  impairs  decision-making  (for  example,  by  presenting  so  much  information  on  so 
many  alternatives  that  it  is  difficult  to  make  a  good  choice). 

The  two  types  of  improper  scoping  are  further  discussed  below. 

Too  Narrowly  Scoped 

Here,  one  or  more  viable  alternatives  have  been  excluded,  inappropriately  limiting 
the  solution  trade  space.  The  most  obvious  way  to  detect  this  problem  is  simply  to  identify 
one  or  more  of  the  missing  alternatives.  If  the  AoA  has  not  provided  a  rationale  for  exclusion 
or  has  simply  assumed  them  out  of  the  picture,  we  can  safely  say  that  the  AoA  was  not  well 
scoped.  If  a  rationale  is  provided,  the  test  is  more  difficult:  namely,  to  demonstrate  that  the 
alternative  might  have  been  preferable  under  some  reasonable  weighing  of  costs  and 
benefits,  notwithstanding  the  stated  rationale. 

Note  that  for  an  AoA  in  progress — or  one  that  has  not  yet  begun — avoiding  an  overly 
narrow  scope  requires  a  way  of  projecting  ahead  to  what  the  costs  and  benefits  might  look 
like  at  the  end  of  the  analysis.  This  fact  argues  strongly  for  taking  an  iterative  approach  to 
defining  alternatives,  and  for  explicitly  tracking  the  upper  and  lower  limits  of  their  likely  costs 
and  benefits  at  every  step  along  the  way.  These  practices  are  reflected  in  the 
recommendations  contained  in  section  of  this  report  titled  Recommendations  and 
Observations.7 

Too  Broadly  Scoped 

Study  teams  operating  under  time  and  resource  constraints  rarely  make  a  deliberate 
choice  to  include  more  alternatives  than  they  can  adequately  analyze  within  those 
constraints.  However,  they  can  miss  opportunities  to  narrow  the  set  of  alternatives,  thus 


5  Note  that  under  these  definitions,  a  particular  AoA  could  be  both  “too  narrow”  and  “too  broad”  at  the  same  time! 
In  other  words,  it  could  inappropriately  exclude  some  alternatives  and  inappropriately  include  others. 

6  The  use  of  the  word  reasonable  here  suggests  some  parallel  with  the  “reasonable  person  standard”  in  tort  law. 
Legal  theorists  disagree  over  whether  the  definition  of  reasonableness  should  be  normative  (that  is,  a  reasonable 
person  is  one  who  does  what  is  cost-effective,  even  if  most  people  would  not)  or  positive  (that  is,  a  reasonable 
person  is  one  who  does  what  most  people  would  do,  even  if  it  is  not  cost-effective;  see  Miller  &  Perry,  2012). 
Fortunately,  these  distinctions  are  not  so  pronounced  when  it  comes  to  choosing  an  AoA.  The  reason  is  that  the 
choice  of  a  solution  path  for  a  DoD  acquisition  is  almost  never  a  completely  objective  determination  because  it 
requires  a  weighing  of  costs  and  non-monetary  benefits  based  on  decision-makers’  values  and  perspectives. 
Hence,  the  definition  in  the  text  boils  down  to  the  question  of  how  likely  it  is  that  one  could  find  decision-makers 
whose  values  and  perspectives  would  have  led  them  to  choose  the  excluded  alternative. 

7  They  are  also  consistent  with  recognized  best  practices  for  screening  alternatives.  See,  for  example,  USAF 
Office  of  Aerospace  Studies  (2008),  AoA  Handbook ,  section  9.1,  which  describes  not  only  initial  screening  for 
nonviable  alternatives,  but  also  “preliminary  screening,”  “later  screening,”  and  “final  selection.” 
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making  less  than  optimal  use  of  the  time  and  resources  available.  Another  possibility  is  that 
AoA  guidance  may  prohibit  such  narrowing  by  specifying  that  certain  alternatives  must  be 
considered  in  the  final  presentation  of  study  results.  Again,  the  opportunity  cost  of  carrying 
these  alternatives  all  the  way  through  the  analysis  may  be  so  great  as  to  impair  the  quality 
of  support  ultimately  provided  to  the  decision-maker. 

Testing  prospectively  (that  is,  before  the  AoA  is  completed)  for  overly  broad  scoping 
is  difficult.  Because  the  range  of  uncertainty  around  the  known  costs  and  benefits  of 
alternatives  generally  starts  out  being  relatively  large,  it  is  easier,  early  on,  to  make  the 
argument  to  leave  an  alternative  in  the  mix  until  the  uncertainties  can  be  reduced  (and  thus 
easier  to  show  that  an  alternative  has  been  prematurely  excluded).  Stated  another  way,  it 
takes  a  certain  expenditure  of  analysis  effort  to  show  that  no  reasonable  decision-maker 
would  be  likely  to  prefer  a  given  alternative. 

Why  Are  Some  AoAs  Poorly  Scoped? 

Our  interviews  with  AoA  consumers,  overseers,  practitioners,  and  critics  disclosed 
several  reasons  why  AoAs  may  be  improperly  scoped. 

Inappropriate  Treatment  of  Time  Constraints 

In  the  section  titled  Background,  we  noted  the  recent  trend  toward  more  stringent 
time  constraints  on  AoAs.  Such  constraints,  perse,  are  not  necessarily  a  problem.  In  fact, 
they  can  contribute  to  better  scoping  by  avoiding  the  situation  in  which  a  protracted  analysis 
and/or  decision-making  process  fails  to  keep  pace  with  fact-of-life  changes  (see  section 
titled  Lack  of  Agility  below).  However,  AoA  stakeholders  may  deal  with  such  constraints  by 
compressing  the  timeline  in  inappropriate  ways.  This  may  occur  before,  during,  or  after  the 
AoA  itself. 

Some  types  of  compression  occur  well  before  AoA  initiation,  during  the  JCIDS 
process.  For  instance,  an  advocate  of  a  particular  new  capability  may  “save  time”  by 
pointing  to  a  prior,  approved  ICD  as  inclusive  of  the  unaddressed  need  for  it.  The  danger  is 
that  the  new  capability  in  question  (for  example,  hold  a  particular  type  of  target  at  risk)  may 
include  a  constraint  (for  example,  with  zero  risk  of  collateral  damage)  that  was  not  analyzed 
in  the  original  capabilities-based  analysis  (CBA).  Because  the  ICD  is  already  approved, 
further  analysis  of  this  constraint  is  not  performed.  As  a  result,  alternatives  that  cause  “only 
a  little”  collateral  damage  are  guaranteed  to  be  excluded  from  the  trade  space.  In  short, 
there  is  no  process  check  on  a  narrowing  of  alternatives  that  has  occurred  even  before  the 
subsequent  AoA  has  begun. 

Immediately  before  or  during  an  AoA,  the  timeline  can  be  compressed  by  summarily 
excluding  certain  alternatives,  without  adequately  considering  their  potential  effectiveness, 
cost,  risk,  and/or  feasibility.  This  action  is  qualitatively  different  from  accelerating  the  rate  at 
which  screening  occurs  within  the  analysis  process:  It  simply  restricts  the  trade  space  based 
on  what  is  assumed  to  be  true.8  In  so  doing,  it  greatly  increases  the  risk  of  an  overly  narrow 
AoA. 


8  A  recent  example:  Starosta  (2013)  cites  a  senior  Air  Force  official  regarding  the  upcoming  “Ground-Based 
Strategic  Deterrent  AoA”:  “[The]  team  is  starting.  We  did  get  re-vectored,  though.  The  department,  in  this 
constrained  budget  environment,  would  like  to  do  those  a  little  faster  with  a  little  less  money,  and  so  they  have 
proposed  a  way  to  streamline  how  they’re  going  to  perform  that  study.”  The  article  goes  on  to  describe  the 
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Finally,  consider  the  case  in  which  an  AoA  study  team  has  used  a  limited  amount  of 
time  to  identify  a  range  of  costs  and  benefits  for  a  set  of  alternatives.  Suppose  the  resulting 
uncertainties  are  such  that  there  is  considerable  overlap  and  no  clear  basis  for  a  decision.  If 
additional  analysis  could  sufficiently  narrow  the  uncertainties,  eliminate  some  of  the 
alternatives,  and  clarify  the  decision,  then  a  refusal  to  extend  the  analysis  timeline  may  be 
considered  an  inappropriate  constraint  that  contributed  to  an  overly  broad  scope  for  the 
prematurely  finished  AoA. 

Inappropriate  Focus  on  Existing  C/CSS  and/or  Operations  Concepts 

Most  interviewees  stated  a  view  that  can  be  paraphrased  as  follows:  “Whenever  I 
see  something  titled  ‘[XYZ]  Replacement  AoA,’  I  automatically  question  whether  the  scope 
may  be  too  narrow.”  The  2009  GAO  study  mentioned  in  the  Background  section  made  a 
similar  point:  an  AoA  for  a  helicopter  replacement,  for  instance,  could  easily  be  incorporating 
a  premature  decision  that  the  underlying  capability  need  is  best  met  by  another  helicopter  of 
some  sort. 

The  danger  can  arise  not  only  from  an  inappropriate  focus  on  existing  C/CSS  (for 
example,  the  notion  that  a  particular  platform  must  be  replaced  by  a  similar  type  of  platform), 
but  also  a  focus  on  existing  concepts  of  operation  (CONOPS).  Consider,  for  example,  a 
sensor  that  provides  input  to  step  1  of  a  two-step  battlefield  decision-making  process.  An 
AoA  that  focuses  exclusively  on  how  well  the  alternatives  support  step  1  could  easily  be 
scoped  improperly,  as  illustrated  in  Figure  2.  Here,  instead  of  a  helicopter  replacement  AoA, 
we  have  what  might  be  termed  the  “Step  1  replacement  AoA.”  In  this  example,  Alternative  A 
would  certainly  be  included  in  the  AoA,  since  it  improves  the  effectiveness  of  step  1  relative 
to  the  base  case.  (Granted,  it  does  not  change  step  2,  but  overall  effectiveness  would  still  be 
improved.)  However,  Alternative  B  might  very  well  be  excluded  unless  the  AoA  scope  was 
widened  to  account  for  the  potential  improvement  it  could  bring  to  step  2. 


proposed  streamlining:  “In  effect,  the  narrower  AOA  will  look  into  fewer  modernization  concepts  than  originally 
planned.”  The  de-scoping  was  reportedly  performed  by  a  SAG  based  on  “data  derived  from  previous  analyses.” 
Note  that  the  authors  are  not  in  a  position  to  judge  the  merits  of  this  particular  de-scoping  or  to  declare  it 
“inappropriate.”  The  point  is  simply  that  time  constraints  can  potentially  exacerbate  the  problem  of  AoA  scoping. 
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Figure  2.  Example  of  Inappropriate  Focus  on  Existing  CONOPS 

Inappropriate  Focus  on  a  Single  Warfighting  Domain 

By  the  time  an  AoA  begins,  a  DoD  lead  component  (most  often,  one  of  the  services) 
has  been  designated  and  has  been  given  responsibility  for  conducting  the  analysis.  That 
component  will  have  played  a  key  role  in  analyzing  the  capability  need  and  developing  the 
ICD;  it  will  also  have  had  to  identify  a  possible  range  of  feasible  alternatives  in  support  of  the 
MDD  (USD[AT&L],  2010).  There  are  many  reasons  behind  the  designation  of  a  DoD  lead 
component:  for  example,  the  availability  of  technology  development  expertise,  personnel, 
models  and  simulations,  and  budget.  Often,  the  nature  of  the  capability  need  documented  in 
the  ICD  makes  the  choice  of  a  lead  service  “obvious.”9  To  explore  this  phenomenon  further 
would  require  a  detailed  look  at  the  role  of  the  services  in  the  various  processes  that  provide 
input  to  the  definition  of  capability  gaps  and  needs  (see  Frittman  et  al.,  2013) — an 
assessment  well  beyond  the  bounds  of  this  study. 

Regardless  of  the  reasons,  service  proponency  of  AoAs  was  identified  by  several 
interviewees  as  a  potential  contributor  to  overly  narrow  scoping.  Colloquially  speaking,  “If 
the  Navy  is  conducting  the  AoA,  the  solution  is  probably  going  to  live  in  the  water;  if  the  Air 
Force  is  conducting  the  AoA,  the  solution  is  likely  to  have  wings;  etc.”  The  problem  is  not 
necessarily  that  one  service  deliberately  sets  out  to  exclude  solutions  that  could  be 
developed  by  another:  each  particular  service  simply  thinks  about  capability  needs  and  their 
solutions  in  a  particular  way,  corresponding  to  the  warfighting  domain  it  represents.  Such 
viewpoints  may  not  lend  themselves  well  to  the  full  exploration  of  solutions  to  joint  or 
coalition  warfighting  needs. 


9  The  WSARA  (2009)  stipulates  an  OSD  check  on  the  extent  to  which  requirements  approved  by  the  Joint 
Requirements  Oversight  Council  (JROC)  have  “engaged  in  consideration  of  issues  of  joint  portfolio 
management”  but  notes  that  this  was  already  required  by  DoD  instruction.  Cf.  Public  Law  111-23,  §201  (c)(3). 
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Inclusion  of  Extraneous  Issues 

The  national  security  landscape  is  such  that  new  concepts  and  ideas  regularly  burst 
into  the  universe  of  discourse:  consider,  for  instance,  the  history  of  low-intensity  conflict, 
information  operations,  or  supply  chain  management.  For  the  purposes  of  this  report,  the 
risk  is  that  a  given  concept  or  issue  may  be  so  much  in  vogue  that  AoA  guidance  mandates 
that  it  be  considered — and  therefore  that  an  alternative  be  included — simply  because  it  is  a 
hot  topic. 

Lack  of  Agility  in  Governance  and  Management  Processes 

As  noted  in  the  Background  section,  a  common  complaint  is  that  both  AoAs  and  the 
JCIDS  process  take  too  long.  Unfortunately,  the  result  is  often  a  reluctance  to  make 
adjustments  during  AoA  execution.  For  example,  even  though  interim  AoA  results  might 
suggest  that  the  underlying  capability  need  should  be  reexamined  or  articulated  differently, 
this  seldom  occurs  because  “too  much  time  has  already  been  expended  just  to  get  to  this 
point.”  One  possible  result:  AoAs  that  are  improperly  scoped  because  they  ignore  fact-of-life 
changes  that  have  rendered  some  alternatives  obsolete  or  made  others  feasible. 

An  associated  problem  is  that  governance  processes  may  not  be  able  to  keep  pace 
with  the  need  for  change.  Suppose  an  AoA  study  team  is  able  to  determine,  during  the 
course  of  an  AoA,  that  the  costs  and  benefits  of  a  given  alternative,  mandated  by  study 
guidance,  will  almost  certainly  cause  it  to  be  dominated  by  the  others.  In  theory,  a  SAG 
could  grant  a  waiver  or  change  of  scope.  However,  if  the  approval  chain  leading  to  the  SAG 
is  such  that  four  months’  worth  of  justification  is  required  to  avoid  three  months  of  wasted 
effort,  the  change  is  unlikely  to  happen. 

Applying  a  Systems  View 

Systems  Thinking  and  the  Conceptagon 

We  used  a  soft  systems  framework,  the  Conceptagon  (Boardman  &  Sauser,  2008), 
to  derive  comparisons  and  characterizations  that  helped  us  better  understand  the  factors 
identified  in  the  section  titled  Why  Are  Some  AoAs  Poorly  Scoped?  The  Conceptagon 
framework  is  organized  around  seven  triplets  (see  Figure  3)  of  system  attributes. 

We  concentrated  on  four  of  these  triplets:10 

•  Wholes,  Relationships,  Parts.  The  identification  of  the  system  at  hand,  the 
constituent  pieces,  and  the  relationships  that  bind  those  pieces. 

•  Structure,  Process,  Function.  The  composition,  arrangement,  or  organization 
(structures)  a  system  employs  to  support  the  key  activities  (processes) 
necessary  to  produce  the  desired  system  behavior  (function). 

•  Inputs,  Transformations,  Outputs.  The  items  that  enter  the  system  (inputs) 
and  exit  it  as  products  or  deliverables  (outputs)  and  the  changes 
(transformations)  that  convert  inputs  to  outputs. 

•  Interior,  Boundary,  Exterior.  The  perimeter  that  separates  entities  that 
comprise  the  system  from  entities  outside  its  control. 


10  The  remaining  triplets  (notably,  Command,  Control,  Communication;  and  Openness,  Hierarchy,  Emergence) 
are  better  suited  to  identifying  organizational  or  governance  solutions,  which  are  not  the  focus  of  this  paper. 
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Figure  3.  Conceptagon  Framework 

(Boardman  &  Sauser,  2008) 

Analysis  of  Selected  Conceptagon  Triplets 
Wholes,  Relationships,  Parts 

What  “system”  are  we  analyzing  when  considering  the  problem  of  AoA  scope? 
Figure  4  identifies  two  very  different  kinds  of  systems  at  work.  The  upper  portion  of  the 
diagram  depicts  what  might  be  termed  the  “AoA  Management  and  Execution  System.”  That 
system  comprises  the  organizational  entities  responsible  for  planning,  conducting, 
documenting,  and  applying  the  AoA.  The  lower  portion  of  the  diagram  depicts  an  entirely 
different  kind  of  “system.”  Specifically,  the  label  “AoA  Object  System”  describes  the  set  of 
C/CSS(s)  that  are  the  object(s)  of  study  in  the  AoA. 
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Figure  4.  The  Two  AoA  Systems  (Prior  to  AoA  Execution) 

In  Figure  4,  the  focus  is  on  the  period  that  precedes  the  AoA  itself.  During  this 
timeframe,  the  Management  and  Execution  System  produces  several  outputs  that  impact 
the  scope  of  the  ensuing  AoA:  documentation  of  the  underlying  capability  need  (in  the  form 
of  an  ICD),  documentation  of  the  MDD  (via  an  Acquisition  Decision  Memorandum11),  and 
formal  AoA  Study  Guidance.  All  these  outputs  are  based  on  the  assessment  of  current  and 
programmed  C/CSSs,  existing  CONOPS  (not  shown),  and  the  resulting  mission  outcomes. 
The  capability  gap  documented  in  the  ICD  is  associated  with  the  particular  set  of  C/CSSs 
that  “should  have  been”  capable  of  producing  the  required  capability.  This  set  will  become 
the  “baseline”  alternative  in  the  AoA.  The  designation  of  the  baseline  establishes  a  boundary 
for  the  Object  System. 

Figure  5  depicts  AoA  execution.  Here,  the  output  of  the  Management  and  Execution 
System  is  the  completed  AoA.  Each  alternative  represents  a  different  possible  instantiation 
of  the  Object  System  and  is  assessed  based  on  its  outputs:  namely,  the  predicted  mission 
outcomes  it  would  produce  or  enable.  The  assessment  may  deem  some  alternatives 
infeasible  on  the  basis  of  these  outcomes;  it  may  also  help  identify  other  alternatives  that 
should  be  considered.  Notice  that  the  reconfigurations  that  give  rise  to  the  alternatives  may 
change  the  definition  of  the  boundary:  a  C/CSS  that  was  formerly  outside  the  boundary  (for 


11  MDD  documentation  can  have  an  important  bearing  on  AoA  scope:  USD(AT&L;  2010)  DTM  10-017  requires 
that  the  MDD  be  based  on  evidence  of  a  range  of  “candidate  materiel  solution  approaches  [that]  have  the 
potential  to  effectively  address  the  capability  gap(s)”  and  are  technically  feasible. 
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example,  the  helicopter  at  the  bottom  right  of  the  collage  in  Alternatives  1  and  2)  may  find 
itself  inside  the  boundary  (the  same  helicopter,  dropping  a  rectangular  object  in  Alternative 
3).  Thus,  the  AoA  scope  is  the  envelope  of  the  boundaries  formed  by  all  the  instantiations  of 
the  Object  System  (that  is,  all  the  alternatives). 


Congress 

-  GAO 


Industry 


/  AoA  Management  and  Execution  System 


Joint  Staff 

-  Staff  elements 

-  FCBs  and  IPTs 

-  JCB 

-  JROC 


Services 

-  Operational  user 

-  Sustaining  org 

-  Developer 

-  Laboratories 

-  Test  centers 

-  Analysis  centers 

Mission  Outcomes 

-  Operational 
effectiveness  of 
Alternative 


%  )  -  jt t 

AoA  Object  System  (Alternative  1)  AoA  Object  System  (Alternative  2)  AoA  Object  System  (Alternative  3) 


Figure  5.  The  Two  AoA  Systems  (During  AoA  Execution) 

In  summary,  AoA  scope  is  a  property  of  a  system  output  (because  an  AoA  is  an 
output  of  the  AoA  Management  and  Execution  System).  It  results  from  one  system  (that  is, 
the  Management  and  Execution  System)  making  decisions  about  the  boundary  of  another 
(that  is,  the  Object  System)  both  before  and  during  the  analysis.  Thinking  of  scope  in  this 
way  allows  us  to  introduce  several  other  principles  of  systems  analysis  (see  section  titled 
Recommendations). 

Structure,  Process,  Function 

An  AoA  lies  at  the  intersection  of  the  three  principal  DoD  decision  support  systems 
(JCIDS;  the  Planning,  Programming,  Budgeting,  and  Execution  System  [PPBES];  and  the 
DAS).  Each  of  the  processes  that  govern  those  three  systems  impacts  an  AoA — and,  in 
particular,  the  actions  taken  by  the  elements  of  the  AoA  Management  and  Execution  System 
to  determine  its  scope.  Figure  6  summarizes  the  key  relationships. 

In  some  cases,  it  is  fairly  obvious  how  these  functions  impact  decisions  regarding  the 
scope  of  an  AoA.  For  example,  resource  availability  can  limit  AoA  scope  directly  (the  fewer 
the  resources  available  for  AoA  execution,  the  fewer  alternatives  can  be  analyzed  to  a  given 
level  of  detail)  or  indirectly  (the  fewer  the  resources  available  for  solutions  to  capability 
needs,  the  fewer  alternatives  will  be  deemed  affordable).  Less  obvious  are  some  of  the 
relationships  we  noted  in  the  Why  Are  Some  AoAs  Poorly  Scoped?  section — for  example, 
the  fact  that  the  set  of  alternatives  is  profoundly  shaped  by  the  way  in  which  a  capability  gap 
is  articulated.  This  latter  point  is  taken  up  in  more  detail  in  Recommendations. 
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Process1 


Function  (relative  to  AoAs) 


Key  Structural 
Elements 


JCIDS 

Assess  current  and  programmed  forces 

Joint  Staff,  JROC 

Identify  capability  gaps  and  needs 

Combatant  commands 
Component  using  commands 

PPBES 

Determine  available  resources  /  affordability 

OSD  CAPE 

Enable  capability  development 

USD(Comptroller) 

Component  HQ 

DAS 

Acquire  future  forces 

USD(AT&L) 

Develop  future  capabilities 

Component  AEs  and 
acquisition  commands 

'The  entries  in  this  column  are  all  termed  "systems";  however,  the  shorthand  reference  is  to  the  processes 
that  govern  the  operation  of  these  systems 


AE  -  Acquisition  Executive  OSD  CAPE  -  Office  of  the  Secretary  of  Defense  Cost  and  Program  Evaluation 

DAS  ~  Acquisition  System  PPOES  -  Planning.  Programming  Hudgeting  and  Execution  System 

JCIDS  -  Joint  Capabilities  Integration  and  Development  System  USD(Comptro*or)  -  Undor  Secretary  of  Defense  (ComptroOor) 

JROC  =  Joint  Requirements  Oversight  Council  USO(AT&l )  -  Under  Secretary  of  Defense  (Acquisition  Technology  &  t  oqrsbcs) 


Figure  6.  DoD  Decision  Support  Functions  That  Impact  AoA  Scope 
Inputs,  Transformations,  Outputs 

Figure  7  depicts  the  notion  that  AoAs  transform  information  about  forces, 
capabilities,  and  C/CSSs  into  information  that  aids  decision-making  by  characterizing  the 
trade  space  of  alternative  solutions.  This  view  extends  the  previous  triplet  by  identifying 
additional  factors  that  impact  AoA  scope.  If  data  and  models  are  not  in  place,  for  example, 
the  resulting  inability  to  analyze  certain  alternatives  within  time  and  resource  constraints 
could  act  to  inappropriately  limit  scope. 


Data 

Models 

Knowledge  of  C/CSSs 
and  technologies 


Cu  rren  t  a  nd  prog  rammed 
forces 

Operations  concepts 
identified  capability  gaps 
and  needs 
Existing  and  planned 
C/CSSs  and 
technologies 


Alternatives  (including 
operations  concepts) 
Completed  AoA: 
information  about 
solution  trade  space 


AoA  study  guidance 
Timelines  /  urgency 
Policy 

Available  resources 


Figure  7.  AoA  Inputs,  Constraints,  Outputs,  Enablers 
Interior,  Boundary,  Exterior 

The  concept  of  AoA  scope  as  a  boundary  on  the  Object  System  was  discussed 
earlier  in  this  section.  The  earlier  discussion  showed  the  boundary  in  what  might  be  termed 
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“C/CSS  space.”  However,  this  boundary  has  other  dimensions  as  well — for  example, 
temporal  dimensions.  Figure  8  shows  that  some  actions  taken  before  AoA  initiation  (those 
on  the  left  side  of  the  figure)  generally  have  broad  impacts  on  AoA  scope  (for  example,  by 
excluding  whole  classes  of  technologies  or  solution  approaches).  Actions  taken  during  AoA 
execution  tend  to  impact  AoA  scope  more  narrowly  (for  example,  making  exclusions  within 
allowable  classes). 


Before  AoA 

During  AoA 

Identification  of  capability  gaps 

Study  pfan  development 

Gap  assessment— CB A 

Characterization  of  alternatives 

Gap  documentation— ICD 

Screening  of  alternatives 

Material  Development  Decisions  DM 

Final  selection  of  alternatives 

AoA  Guidance 

CRA  i  CapflhiliNfts-Based  Assfissjt^nt 

ICD  -  InibBt  Capabilities  Doctimonl 

ADWI  -  Asqurjibge  D«Cf$H?n  Men  iyr  anti  urn 

Figure  8.  Timeframe  of  Actions  That  Impact  AoA  Scope 

Recommendations  and  Observations 

The  study  team  combined  insights  based  on  the  system  formulation  of  the  problem 
(see  the  section  titled  Applying  a  Systems  View)  with  insights  gained  during  interviews  (see 
the  section  titled  Understanding  the  Problem)  to  arrive  at  a  series  of  recommendations, 
which  are  presented  in  below,  followed  by  some  concluding  observations. 

Recommendations 

The  guiding  principles  that  follow  are  intended  to  help  participants  in  the  AoA 
Management  and  Execution  System  reduce  the  incidence  of  improperly  scoped  AoAs. 

Focus  on  Outputs  and  Think  Backwards 

An  alternative  is  any  potential  configuration  of  C/CSSs  that  could  produce  the 
required  output(s):  namely,  the  military  outcomes  that  would  address  the  documented 
capability  gap  to  some  degree.  In  formulating  alternatives,  it  is  tempting  to  think  forwards — 
that  is,  to  envision  a  particular  set  of  C/CSSs  and  determine  whether  it  produces  the 
required  outcome(s)  (or  could  be  modified  or  reconfigured  to  do  so). 

To  some  extent,  the  JCIDS  process  requires  participants  to  think  in  terms  of 
particular  C/CSSs  during  the  formulation  and  analysis  of  capability  gaps.  Sponsors  of 
capability  gaps  are  directed  to  consider  whether  “capability  solutions  which  can  satisfy  the 
Sponsor  capability  requirements  exist  elsewhere  in  the  Joint  force  [emphasis  added]” 

(CJCS,  2012,  para.  5).  And  CBAs  require  “the  operational  assessment  of  the  current  and 
programmed  force  [emphasis  added]”  (CJCS,  2012,  para.  2). 

Consideration  of  C/CSSs  during  gap  formulation  is  actually  inevitable:  It  may  be 
possible  to  think  about  a  capability  without  reference  to  the  system  that  provides  that 
capability;  however,  it  is  not  possible  to  think  about  a  capability  gap  without  reference  to  one 
or  more  sets  of  C/CSSs  that  fails  to  do  so.  Thus,  by  the  time  an  MDD  is  reached  and  an  ICD 
generated,  the  Object  System — including  its  boundaries  and  interfaces — has  already  been 
conceived  ...  with  reference  to  the  current  and  programmed  force.  The  danger  is  that  this 
process  may  induce  tunnel  vision  and  drive  the  AoA  to  an  overly  narrow  scope. 


MSi 


-457- 


ACQUISITION  RESEARCH  PROGRAM: 
CREATING  SYNERGY  FOR  INFORMED  CHANGE 


How  does  one  escape  this  dilemma?  At  some  point  before  the  formulation  ofAoA 
study  guidance,  there  should  be  an  attempt  to  work  backward  from  a  (C/CSS-neutral) 
statement  of  the  required  capability  to  a  set  of  feasible  alternatives  that  could  potentially 
satisfy  it.  Such  an  attempt  should  not  be  bound  in  any  way  by  the  thinking  contained  in  the 
CBA. 


Formal  systems  analysis  tools  and  methods  can  help  in  drawing  the  necessary 
linkages.  These  methods  also  show  clearly  the  difference  between  the  “forward”  and 
“backward”  thought  processes  alluded  to  above.  Within  version  2.0  of  the  DoD  Architecture 
Framework  (DoDAF),  for  example,  the  “forward”  progression  of  thought  corresponds  to  the 
movement  from  Operational  View  (OV)  to  Capability  View  (CV).12  It  asks,  “What  is  the  CV 
associated  with  this  particular  OV?”  This  process  may  be  used  during  the  assessment  of 
current  and  programmed  forces  to  identify  capability  gaps,  resulting  in  an  ICD. 

What  we  have  termed  the  “backward”  thought  progression  can  be  represented  by  a 
subsequent  movement  from  CV-1  (Capability  Vision)  to  CV-6  (Capability  to  Operational 
Activities  Mapping),  as  shown  in  Figure  9.  The  CV-1  essentially  captures  the  sense  of  the 
ICD.  Forcing  the  analyst  to  carefully  translate  capabilities  to  activities,  it  becomes  possible 
again  to  work  from  a  clean  sheet  of  paper  by  asking  questions  such  as  “Are  there  other 
kinds  of  activities  or  combinations  of  activities  that  could  produce  these  capabilities?”  “How 
might  such  activities  be  performed  or  linked  to  do  that?”  In  the  end,  one  is  asking,  in  DoDAF 
terms,  “What  kinds  of  OVs  could  correspond  to  this  CV?”  Or,  in  other  words,  “What  kinds  of 
C/CSSs,  interacting  with  each  other  and  with  the  user  in  what  ways,  could  potentially  meet 
the  stated  need?” 


Figure  9.  Translating  Capabilities  to  Activities:  Using  DoDAF  v2.0 

(Wayson,  2010,  slides  9  and  14)13 


12  Note  that  DoDAF  itself  does  not  prescribe  a  progression  of  thought.  In  applying  DoDAF,  one  can  start  almost 
anywhere,  producing  only  the  views  required  for  a  particular  purpose. 

13  In  this  example,  the  CV-6  includes  capabilities  not  shown  in  the  CV-1.  The  additional  capabilities  implicit  in  the 
CV-1  were  formally  identified  in  the  Capability  Taxonomy  (CV-2). 
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The  “backwards”  thought  transition  from  CV  to  OV  has  an  additional  benefit.  By 
setting  up  the  question  of  user  interaction  with  the  system,  it  stimulates  additional  thinking 
about  the  Object  System  boundary  and  how  that  boundary  contributes  to  the  definition  of 
alternatives  (see  the  next  subsection). 

Start  With  the  Exterior  and  Work  Inwards 

The  usual  thought  process  is  to  start  with  a  collection  of  C/CSSs  believed  to 
represent  an  instantiation  of  the  Object  System  and  move  conceptually  “outwards.”  Those 
entities  without  which  the  Object  System  could  not  exist  or  function — provided  they  are 
within  the  decision-maker’s  sphere  of  influence  and  authority — are  deemed  to  lie  in  the 
interior;  all  other  entities  are  in  the  exterior.  Although  this  may  be  a  sound  method  of  refining 
an  initial  definition,  it  may  not  be  the  best  approach  to  identifying  alternative  instantiations  of 
the  Object  System. 

The  suggested  approach  is  to  reverse  the  process — that  is,  to  start  by  identifying 
entities  that  are  clearly  not  part  of  the  system  and  then  moving  conceptually  “inwards.”  If  the 
stated  need  is  to  improve  the  autonomous  navigation  capabilities  of  an  existing  fleet  of 
aircraft,  for  example,  it  may  be  easy  to  determine  that  alternatives  involving  entirely  new 
airframes  are  unaffordable.  The  airframes  themselves  now  lie  in  the  exterior  of  the  Object 
System  and  impose  a  set  of  compatibility  constraints.  However,  it  may  not  be  clear  whether 
alternatives  that  upgrade  both  navigation  and  fire  control  systems  are  cost-effective.  The 
“outside-in”  approach  might  allow  both  types  of  solutions  among  the  potential  alternatives  in 
an  AoA.14 

Outside-in  exploration  of  the  boundary  also  helps  identify  interfaces  that  “[participate] 
in  constituting  the  system”  (Cilliers,  2005,  p.  611).  The  eardrum,  for  example,  “forms  the 
boundary  between  the  inner  and  the  outer  ear,  but  at  the  same  time  it  exists  in  order  to  let 
the  sound  waves  through”  (Cilliers,  2005,  p.  611).  In  the  same  way,  the  interface  constraints 
identified  during  an  outside-in  exploration  of  the  Object  System  boundary  point  toward 
important  characteristics  of  AoA  alternatives.  They  can  represent  more  than  simply 
feasibility  constraints.  For  example,  they  can  uncover  important  measures  of  effectiveness. 

Apply  Constraints  Carefully 

Every  constraint  imposed  and/or  accepted  constitutes  a  decision  to  exclude  some 
portion  of  the  solution  trade  space  from  further  consideration.  Accordingly,  it  should  be 
supported  by  some  testable  rationale  (that  is,  an  explanation  of  what  portion  of  the  trade 
space  is  being  excluded  and  why  it  is  not  worth  considering).  An  important  question  to  ask: 
What  would  it  take  to  change  this  constraint?  To  see  the  importance  of  asking  this  question, 
consider  several  examples. 

The  first  example  considers  the  well-known  constraint  termed  affordability.  For  many 
years,  decision-makers  relied  on  AoAs  to  identify  cost-effective  solution  approaches  only  to 
discover  that  the  resulting  programs  could  not  be  executed  with  available  resources — that  is, 
they  were  unaffordable.  In  response,  the  USD(AT&L)  has  mandated  that  affordability  be 
considered  at  Milestone  A  (Husband  &  Kaspersen,  2012,  p.  9).  An  overly  restrictive 
approach  would  be  to  define  affordability  as  the  difference  between  expected  out-year 


14  An  interesting  parallel  in  the  world  of  object-oriented  software  design  is  the  recent  interest  in  “Outside-In 
Development.”  For  a  practical  illustration,  see  Bache  (2013). 


MSi 

TMlBWlTt*  HH 


ACQUISITION  RESEARCH  PROGRAM: 
CREATING  SYNERGY  FOR  INFORMED  CHANGE 


-459- 


obligational  authority  and  the  expected  out-year  costs  of  current  and  programmed  C/CSSs. 
That  approach,  however,  fails  to  consider  the  possibility  that  the  benefits  of  meeting  the 
capability  need  at  hand  may  be  so  great  as  to  be  worth  terminating  or  forgoing  some  current 
or  programmed  C/CSSs.  Rather  than  unduly  constrict  the  trade  space  with  this  narrow 
definition,  decision-makers  would  be  better  served  by  asking,  “What  other  capabilities  would 
we  be  willing  to  forgo  to  achieve  this  one?”15 

Another  example  was  mentioned  earlier:  The  Object  System  is  usually  constrained  to 
exclude  elements  that  are  outside  the  decision-maker’s  sphere  of  influence  or  authority  (see 
Oliveira,  1973,  p.  3).  Suppose  the  decision-making  authority  is  organization  X  and  all 
materiel  alternatives  are  believed  to  be  related  to  technology  A,  for  which  that  organization 
is  responsible.  Further,  suppose  that  the  AoA  study  team  discovers,  while  developing  its 
study  plan,  a  set  of  potential  alternatives  based  on  technology  B,  for  which  the  responsible 
organization  is  Y  (a  parallel  organization  in  the  hierarchy).  The  argument  that  “those  aren’t 
our  technologies  and  we  can’t  make  decisions  about  them”  should  not  be  considered 
sufficient  grounds  for  dismissing  the  newly  discovered  set  of  alternatives — even  though  the 
consequences,  in  terms  of  process,  may  be  painful  (for  example,  moving  the  decision¬ 
making  authority  to  a  higher  level). 

Even  the  capability  gap  itself  can  be  articulated  in  such  a  way  as  to  over-constrain 
the  solution  trade  space — as,  for  example,  might  occur  if  alternatives  that  fail  to  close  the 
gap  completely  are  dropped.  During  the  interviews  the  study  team  conducted,  we  learned 
that  new  JCIDS  guidance  under  development  will  address  this  point  by  stating  clearly  that 
ICDs  should  not  include  threshold  values  for  key  performance  parameters. 

Iterate  and  Reduce  Uncertainty 

Because  the  environment  can  change  over  time,  the  boundary  of  the  Object  System 
should  be  allowed  to  change  as  well.  Standard  AoA  practice  accounts  for  such  change 
through  the  normal  screening  process.  Less  common,  but  equally  important,  is  the 
occasional  need  to  add  alternatives  or  fundamentally  change  the  nature  of  the  alternative 
set.  Such  changes  can  arise  for  a  number  of  reasons:  for  example,  during  the  interval 
between  the  CBA  and  the  AoA,  the  state  of  technology  may  have  evolved  such  that 
alternatives  once  thought  to  be  infeasible  are  now  feasible. 

The  recommendation  is  to  take  an  iterative  approach:  specifically,  to  check  and 
recheck  the  validity  of  assumptions  and  constraints  throughout  AoA  preparation  and 
execution.  Note  that  this  concept  extends  to  the  period  before  the  AoA  has  actually  begun. 

In  the  earlier  example  of  the  aircraft  navigation  system  (see  the  above  subsection  Start  With 
the  Exterior),  we  remarked  that  a  certain  decision  might  be  “easy  to  determine”  and  that 
another  decision  “might”  result  in  a  certain  outcome.  Ideally,  these  decisions  are  informed  by 
quick  analysis  at  a  low  level  of  detail.  Although  the  analysis  results  may  come  with  a  wide 
range  of  uncertainty,  they  may  still  be  sufficient  to  allow  for  the  right  decision  regarding  the 
Object  System  boundary.  If  not,  the  analysis  is  refined  to  the  point  that  it  does  support  the 
decision. 


15  Based  on  the  interviews  the  study  team  conducted,  there  is  evidence  that  this  question  is  now  being  posed  as 
a  way  of  attaching  resource  priority  levels  to  DoD  capability  needs. 
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The  associated  need  is  for  CBA  and  AoA  governance  processes  that  can  respond 
quickly  to  the  need  for  such  adjustments.  The  establishment  of  the  SAG  (supported  by  Joint 
Staff  Functional  Control  Boards)  as  an  ongoing  governance  mechanism  is  undoubtedly  a 
step  in  the  right  direction — although  there  is  evidence  from  the  study  team  interviews  that 
the  process  is  not  yet  as  agile  as  it  needs  to  be.  For  example,  it  is  not  clear  to  what  extent 
the  SAG  can,  on  its  own  authority,  redirect  the  execution  of  an  AoA  along  lines  that  deviate 
from  the  boundaries  established  by  a  validated  ICD.  If  the  impacts  of  real-world  change  can 
be  accommodated  only  by  restarting  the  JCIDS  process  from  scratch,  they  are  unlikely  to  be 
considered  at  all. 

Summary  and  Conclusions 

A  scheme  for  applying  these  recommendations  is  summarized  in  Figure  10,  which 
shows  the  alignment  of  key  principles  according  to  stakeholder  roles  and  process  timelines. 
The  absence  of  an  entry  does  not  mean  that  participant  is  exempt  from  that  particular 
principle — merely  that  it  is  not  necessarily  key  to  AoA  scoping  at  that  particular  point  in  the 
AoA  process. 
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Figure  10.  Summary  of  Key  Principles  for  Improving  AoA  Scoping 

One  important  implication  of  Figure  10  is  that  systems  thinking  should  not  be 
considered  an  exercise  solely  for  AoA  study  team  analysts.  Senior  executives,  warfighters, 
and  DoD  business  process  owners  can  all  benefit  from  applying  a  systems  view  to  the 
practice  of  AoAs.  That  practice  continues  to  evolve  within  the  DoD,  and  the  challenges  of 
the  future  will  doubtless  be  different  from  those  of  the  past.  Cultural,  organizational,  and 
individual  behaviors  will  continue  to  frustrate  the  best  process  designs.  Nevertheless,  the 
study  team  is  confident  that  the  above  principles  can  improve  the  DAS  by  reducing  the 
incidence  of  poorly  scoped  AoAs  and  thus  enable  better  acquisition  outcomes. 
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Cyclic  Use  of  Prototyping 


•  "Fly-Before-Buy" 


Weapons  System  Acquisition  Reform  Act  of  2009 

and  Prototyping 


•  Competitive  prototyping  of  systems  or  critical 
subsystems  before  Milestone  B  approval 

•  If  competitive  prototyping  is  waived  by  MDA,  a 
prototype  must  still  be  produced  before  MS  B 
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Mandatory  for  MDAPs 


PDR  Before  Milestone  B 

PDR  After  Milestone  B 

•  Planned  for  in  Acquisition  Strategy 
•PDR  Report  provided  to  MDA  at  MS  B 
•Includes  recommended  requirements  trades 
resulting  from  prototyping  and  critical 
technology  demonstrations 
•Mandatory  for  MDAPs  and  DASD(SE) 
participates 

•Planned  for  in  Acquisition  Strategy 
•PDR  Report  provided  to  MDA  prior  to  Post  PDR 
Assessment 

•Reflects  requirements  trades 
•At  Post  PDR  Assessment,  MDA  considers  PDR 
report;  determines  action(s)  required  to  achieve 
APB  objectives  and  issues  ADM 
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Section  2366b  of  Title  10,  United  States  Code,  requires  certification  that  the  technology  in  the  program  has 


been  “demonstrated  in  a  relevant  environment”  prior  to  IVSIestone  B.  (This  is  interpreted  as  TRL  6.) 


Joint  Lightweight  Tactical  Vehicle  (JLTV) 


USA  /  USMC 
Contract  Type 
TD  Contract  Costs 
Requirements 
TMRR  Phase 
Prelim  Design  Rev 
TRL  (at  MS  B) 


BAE  Systems 

AM  General/GDLS 

Lockheed  Martin 

Various 

Various 

CPFF 

$62.9  M 

$61.3  M 

CDD,  15  March  2012 

27  months 

June  -  July  2009 

$53.4  M 

5  (underbody)  /  TD  protoypes  built  on  assembly  line 

Littoral  Combat  Ship  (LCS) 


USN  General  Dynamics  Lockheed  Martin 


Contract  Type 
TD  Contract  Costs 
Requirements 
TMRR  Phase 

Preliminary  Design  Review 
TRL  (at  MS  B) 


FPI  FPI 

$575  M  $537  M 

validated  CDD,  June  2008;  10  KPPs 
72  months 

July  2003  (prior  to  MS  A) 

?  (integration  w/mission  packages)  /  9  (seaframe) 


Small  Diameter  Bomb  (SDB)  II 


USAF  /  USN 
Contract  Type 
TD  Contract  Costs 
Requirements 
TMRR  Phase 
Critical  Design  Rev 
TRL  (at  MS  B) 


Boeing  /  Lockheed 

CPFF 
$161.4  M 


Raytheon 

CPFF 
$161.4  M 


validated  CDD,  June  2009;  5  KPPs 
42  months 

within  6  months  of  MS  B  (June  2010) 
6  (Program  Office  Estimates) 


Research  Issue 


Determine  if  DoD  Instruction  5000.02  policies  for  Major 
Defense  Acquisition  Programs  (MDAPs)  relating  to 
competitive  prototyping,  technology  readiness,  and 
Preliminary  Design  Review  (PDR)  prior  to  Milestone  (MS)  B 
are  having  the  desired  effect  on  program  outcomes. 

Research  questions: 

1.  Does  the  knowledge  from  competitive  prototyping  and 
a  PDR  conducted  prior  to  MS  B  result  in  better  decisions 
relative  to  requirements,  design,  and  resources? 

2.  What  are  the  effects  of  the  competitive  prototyping, 

technology  readiness,  and  PDR  policies  on  program  costs 
and  program  schedules?  9 


Research  Methodology 

•  Cost  growth  was  determined  by  comparing  the  original  Program 
Acquisition  Unit  Cost  (PAUC)  with  the  current  PAUC  estimate, 
calculated  to  the  same  base-year  dollars,  as  reported  in  the  Unit 
Cost  Report  (UCR)  of  the  annual  Selected  Acquisition  Reports 
(SARs)  for  2011  and  2012 

•  Annual  SARs  also  identify  if  programs  have  suffered  an  Acquisition 
Program  Baseline  (APB)  threshold  schedule  breach 

•  Government  Accountability  Office  (GAO)  survey  data  was  used  to 
identify  programs  that  have  demonstrated  technology  maturity  on 
prototypes  in  a  relevant  environment  (Technology  Readiness  Level 
6)  and  have  conducted  a  preliminary  design  review  prior  to 
Milestone  B 
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Life  Cycle  Cost  Composition 


Research  Methodology 

Descriptive  statistics  are  used  to  analyze  cost  growth  (percent 
change  to  date  in  PAUC)  and  schedule  breaches  for  the  MDAPs  that 
have  conducted,  competitive  prototyping  and  PDR  activities. 

Similar  descriptive  statistics  are  used  to  analyze  the  balance  of_ the 
MDAPs  included  in  a  particular  annual  SAR  submission. 

The  percentage  of  programs  that  have  negative  cost  growth 
(negative  percent  change  to  date  in  PAUC)  from  each  population  is 
compared.  The  population  with  the  highest  number  of  negative 
cost  growth  programs  is  preferred. 

The  percentage  of  programs  that  suffered  an  APB  schedule  thres¬ 
hold  breach  from  each  population  is  compared.  The  population 
with  the  lowest  percent  of  schedule  breaches  is  preferred.  n 


Research  Results 


PAU£  Cost  Growth  Results.  Based  upon  data  from 
the  2011  and  2012  SAR,  programs  that 
demonstrated  technology  maturity  on  prototypes 
in  a  relevant  environment  (TRL  6)  and  conducted 
a  preliminary  design  review  prior  to  Milestone  B 
were  rriore  often  to  show  negative  PAU£  cost 
growth. 

This  result  was  seen  in  all  DoD  Components. 
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Research  Results 


Table  2.  Programs  Costing  Less,  Selected  Acquisition  Report,  December  31 , 201 1 


Component 

Programs  w/Prototypes  &  PDR 

Balance  of  Programs 

Programs 

Costing 

Less 

Total 

Programs 

Percent 

Programs 

Costing 

Less 

Total 

Programs 

Percent 

Army 

6 

7 

86 

3 

12 

25 

Navy 

7 

15 

47 

6 

20 

30 

Air  Force 

5 

10 

50 

4 

15 

27 

Def  Agency 

1 

1 

100 

2 

9 

22 

Total 

19 

33 

57 

15 

56 

27 

Table  3.  Programs  Costing  Less,  Selected  Acquisition  Report,  December  31, 2012 


Component 

Programs  w/Prototypes  &  PDR 

Balance  of  Programs 

Programs 

Costing 

Less 

Total 

Programs 

Percent 

Programs 

Costing 

Less 

Total 

Programs 

Percent 

Army 

5 

8 

62 

4 

12 

33 

Navy 

8 

18 

44 

4 

20 

20 

Air  Force 

3 

8 

38 

6 

17 

35 

Def  Agency 

0 

0 

0 

4 

5 

80 

Total 

16 

34 

47 

18 

54 

33 

14 


Research  Results 


Schedule  Threshold  Breach  Results.  Based  upon 
data  from  the  2011  and  2012  SAR,  programs  that 
demonstrated  technology  maturity  on  prototypes 
in  a  relevant  environment  (TRL  6)  and  conducted 
a  preliminary  design  review  prior  to  Milestone  B 
did_  not  suffer  fewerAPB  schedule  threshold- 
breaches. 

This  result  was  seen  in  all  DoD  Components 
except  the  Air  Force. 
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Research  Results 


Table  4.  Program  Schedule  Breach,  Selected  Acquisition  Report,  December  31 , 201 1 


Component 

Programs  w/Prototypes  &  PDR 

Balance  of  Programs 

Programs 

w/Schedule 

Breach 

Total 

Programs 

Percent 

Programs 

w/Schedule 

Breach 

Total 

Programs 

Percent 

Army 

2 

7 

28 

2 

12 

17 

Navy 

4 

15 

27 

5 

20 

25 

Air  Force 

4 

10 

40 

6 

15 

40 

Def  Agency 

1 

1 

100 

4 

9 

44 

Total 

11 

33 

33 

17 

56 

30 

Table  5.  Program  Schedule  Breach,  Selected  Acquisition  Report,  December  31, 2012 


Component 

Programs  w/Prototypes  &  PDR 

Balance  of  Programs 

Programs 

w/Schedule 

Breach 

Total 

Programs 

Percent 

Programs 

w/Schedule 

Breach 

Total 

Programs 

Percent 

Army 

3 

8 

38 

4 

12 

33 

Navy 

6 

18 

33 

3 

20 

15 

Air  Force 

2 

8 

25 

7 

17 

41 

Def  Agency 

0 

0 

0 

0 

5 

0 

Total 

11 

34 

30 

14 

54 

26 
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Future  Research 


•  To  remove  some  of  the  uncertainty  in  the  cost  growth  metric,  compare  PAUC 
based  upon  the  original  cost  estimate  with  actual  PAUC.  Actual  PAUC  can  be 
determined  from  contracts  found  in  the  Defense  Cost  and  Resource  Center 
(DCARC)  database. 

•  To  remove  some  of  the  uncertainty  in  the  schedule  slippage  metric,  compare  the 
original  schedule  estimate  with  actual  schedule  performance  data.  Actual 
schedule  performance  data  for  this  comparison  should  be  available  in  the  DCARC 
database  or  Defense  Acquisition  Management  Information  Retrieval  (DAMIR). 

•  Finally,  the  challenge  in  using  cost  growth  and  schedule  slippage  metrics  is  to  tie 
them  back  to  the  use  of  competitive  prototyping  (to  reveal  technology  readiness) 
and  the  use  of  an  early  PDR.  The  knowledge  from  these  activities  and  how  that 
knowledge  is  applied  will  tell  us  whether  these  policies  have  had  an  effect.  To 
that  end,  more  detailed  surveys,  such  as  those  conducted  annually  on  selected 
weapon  systems  by  the  GAO,  will  aid  in  helping  establish  the  cause-effect 
relationship  between  policy  and  program  outcomes. 
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